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I. Overview 



In previous work (Chaplin and Capizzano 2006), we conducted an evaluation of a summer 
learning intervention and found it to be effective using the gold standard of research methods, 
random assignment. Given those positive results, this new paper describes the specific elements 
of the successful program so it can be replicated, and investigates potential barriers to 
implementation and replication. As with most evaluation studies our study only estimated 
impacts of the program overall. This means that we cannot distinguish which elements caused 
the positive impacts. However, we can say that the elements described here were present in this 
successful program and thus of potential importance. 

This observation and interview-based process study describes activities that occurred during the 
summers of 2004 and 2005 in an academically-oriented summer program that receives both 
federal and private funding: the Building Educated Leaders for Life (BELL) Accelerated 
Learning Summer Program. The BELL program is an innovative, academic summer program 
with a strong, asset-based youth development approach. It employs well-developed curricula in 
both reading and math and contains features of positive developmental settings outlined by the 
National Research Council and Institute of Medicine in the report. Community Programs to 
Promote Youth Development (2002). Most importantly, as noted above, the program has 
demonstrated effectiveness in reducing summer learning loss among low-income children based 
on both a rigorous random assignment external evaluation (Chaplin and Capizzano 2006) and 
two internal evaluations (T. Cooper 2002, 2003). 

This report serves two purposes. Lirst, it allows researchers to examine whether BELL’s program 
components are implemented with fidelity (i.e., as they said they would be in program 
documents). This examination allows the research team to better understand the nature of the 
BELL intervention in practice and assist with the interpretation of outcome study results. 

Second, this report describes implementation issues that may affect whether the BELL program 
can be replicated in other sites. Issues that affect program replication often include difficulty in 
obtaining sustainable funding, high-quality program staff, and physical space, among others. In 
addition, it is also important to understand if there are any unique characteristics associated with 
the existing BELL sites that may not be easily reproduced. These characteristics may include an 
abundance of partners in the community from which to draw resources, a lack of competing 
programs, or other environmental characteristics that help to support the program. 

This process report was developed as part of a larger study that included a random assignment 
impact evaluation of this summer program. Students were randomly assigned to treatment and 
control groups. Only the treatment group was given access to the BELL summer program. 
Impacts were estimated by comparing outcomes for the treatment and control groups. 

Random assignment was done in both the summers of 2004 and 2005. However, student survey 
response rates were very low in 2004 so the impact report cited above is based on the 2005 data. 
This background is important for two reasons. Lirst, random assignment in 2004 had some 
impacts on program operations, as noted below. Second, the impact results reported above are 
based on the 2005 program, which differed somewhat from the 2004 program. This report 
describes both the 2004 and 2005 BELL summer programs and how they differed. 
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II. Background 

Over the past decade, an emerging body of research has shown that low-income children 
experience a decline in academic progress during the summer months. While higher-income 
children may also experience some decline, the test scores of low-income children drop 
significantly both in absolute terms and relative to their higher-income counterparts during the 
summer (Entwisle and Alexander 1992; Cooper et al. 1996). This evidence suggests that income 
differences are related to a disparity in the opportunities that children have to hone and maintain 
their academic skills over the summer months. Perhaps more importantly, this learning loss 
likely contributes to the dramatic overall differences in long-term educational achievement 
between low- and higher-income children. 

In recent years, a significant amount of public funding has been used to provide enriching 
summer programs for low-income children. For instance, a number of 21st Century Community 
Learning Center (CCLC) grantees use their federal funds to run summer programs. Title I funds 
may also be used to fund summer programs. Moreover, additional funds for summer programs 
are available because of provisions in the new federal education legislation, the No Child Left 
Behind Act of 2001, which sets aside Title I funds for supplemental education services (SES) for 
students in low-performing schools. These funds can be used to fund summer programs that are 
approved by the state education agency, though in practice most SES funds are used for school 
year activities. 

III. Process Study Methods 

The findings from this process study are based on data collected from program documents, a 
phone interview with the BELL CEO, and site visits to the BELL sites taking part in our study. ^ 
During the site visits, site managers were interviewed, focus groups were held with teachers and 
teaching assistants, and structured observations of a randomly-selected sample of classroom 
lessons were conducted. We discuss these methods in detail below. 

Review of Program Documents 

In order to determine whether the BELL program was implemented with fidelity, program 
materials including informational booklets, parent guides, summer schedules, and internal 
evaluation documents were obtained and reviewed to gain a strong working knowledge of the 
day-to-day operations of the program. In addition, sample materials from the BELL 
reading/writing and math curricula were obtained, along with teacher lesson plans for the day of 
each site visit. These documents were used to describe the overall goals and administration of the 



* In the summer of 2004, we visited one site in New York (PS 125) and one site in Boston 
(Trotter Elementary school). In the summer of 2005, we visited a different site in New York City 
(PS 33) and in two sites in Boston (Trotter and Greenwood Elementary schools). The program 
also operated in a few other sites during these years. These additional sites were not included in 
our study either because of commitments to school principals that a targeted group of students 
would be served or, in one case, because some students who had been long-term participants in 
BELL had been promised that they could participate in the summer program. 
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program, as well as to develop observation and interview protocols to determine whether the 
program components and curricula were implemented as intended. 

Semi-structured Interviews with the BELL CEO and Site Managers 

To obtain information from senior staff about the implementation of the BELL program, semi- 
structured interview protocols were developed for a phone interview with the BELL CEO and in- 
person interviews with BELL site managers. These protocols were designed to collect 
information on some of the broad implementation issues associated with the program, including 
the acquisition of the physical space to house the program, staffing issues, funding issues, and 
partnerships with other community organizations. These respondents were also asked about the 
prevalence of competing programs in the community, as well as about major benefits of the 
BELL program that could potentially go unmeasured by the outcome study. 

Focus Groups of BELL Teachers and Teaching Assistants 

Focus group protocols were developed to collect data from teachers and teaching assistants about 
the day-to-day implementation of the BELL program. The participants were asked a range of 
questions including their assessment of the general administration of the program, the curricula 
used, and issues related to the varying skill levels of the students, behavior management, and 
parental involvement. 

Classroom Observations 

Two observational instruments were developed to record teacher and student activities for 
randomly selected BELL lessons. The first observation instrument recorded teacher activities 
within the classroom and how teachers dealt with common issues like varying skill levels, lack of 
engagement, and student misbehavior. 

The second instrument was an observational matrix designed to record student activities at 10- 
minute intervals throughout the class period. Trained observers recorded the percentage of 
students in class doing one of the following activities at each time interval: listening to the 
teacher lecture; working independently; working in groups; participating in a class activity; 
daydreaming, otherwise disengaged, or misbehaving; and other. These protocols were used to 
observe the math and reading classrooms in the morning and the enrichment activities in the 
afternoon. 
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IV. Findings 



BELL In Theory: The Program on Paper 
Overview 

The BELL summer program is designed as a six-week accelerated learning program operating 
Monday-Lriday from 8:30 a.m. to 4:30 p.m. The overarching goal of the program is to 
“dramatically increase the academic achievements, self-esteem and life opportunities of 
elementary school children living in underserved communities....” (T. Cooper 2003, pg 1). 
bell’s philosophy is to serve the whole child: it aims to not only increase academic success by 
improving basic math and literacy skills, but also works to assist in social and emotional 
development by exposing program participants to positive role models, and by building self 
esteem and encouraging parents to become more involved in their children’s lives. 

In both summers, the program operated from Tuesday after July 4th through mid- August. In 
2004, the program ran for a total of 25 days in one site and 27 in the other. In 2005, the program 
ran for a total of 28 days at two sites and 29 at a third. In both years, Monday through Thursday 
consisted of academic instruction in the morning and enrichment activities in the afternoon. On 
Lridays, students, referred to as scholars, listened to guest speakers from the community in the 
mornings and in the afternoons took field trips that were designed to teach the importance of 
community involvement. 

A Typical Week 

Monday-Thursday, scholars spend their mornings clustered by grade in classrooms normally 
taught by one teacher and one teaching assistant (most often a college student pursuing a career 
in educationj. Class sizes have a low student to teacher ratio — 1 teacher and 1 TA for every 15 
students as reported by BELL (T. Cooper 2002).^ Each week students are scheduled to receive 
approximately eight hours of literacy (reading and writing) instruction (two hours per day, four 
days per week), four hours of math instruction (one hour per day, four days per week), 12 hours 
of various enrichment activities, such as art, music, dance, and gym time (three hours per day, 
four days per week), and two hours of community time (thirty minutes per day, four days per 
week). 

bell’s afternoon activities include music, dance, art, theatre and physical education. Depending 
on the site, the scholars rotate to either two or three different activities during the afternoon 
where they work on projects that culminate at the end of the 6-week program. Lor example, in 
the dance class, each group will work on a particular routine throughout the summer to be 
performed at the closing ceremonies. Lurthermore, each enrichment activity is curriculum-based 
and contains educational components. 



^ In May 2007, BELL staff informed us that in 2004 and 2005 their goal was to have 16 to 18 
students in each class with two adults. 
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On Fridays, program participants listen to a guest speaker in the morning and go on field trips in 
the afternoon. Guest leeturers are members of the local community and their presentations are 
intended to make scholars aware of the opportunities afforded by education. The guest speaker 
may be a loeal author, musieian, professor, or artist. The program seeks to teaeh ehildren the 
importanee of eommunity involvement and to also show the eommunity the promise of the 
BELL participants. The field trips serve to expose the children to cultural resources and fun 
activities within or nearby their communities. In the past, students have visited aquariums, 
museums, professional sports games, and historie sites. 

Curricula 

BELL ehanged their math and reading eurrieula during the eourse of our study. In both years the 
math and literaey interventions were based on nationally reeognized eurrieula. In 2004, the 
reading and writing portion were drawn from a culturally sensitive curriculum developed by 
Voices for Love and Lreedom (VLL), a nonprofit educational organization affiliated with the 
New Ameriean Sehools. This eurrieulum was designed to explore various themes, sueh as 
demoeraey, through the use of multieultural literature. In addition, this eurrieulum was aligned 
with several national and state learning standards as well as some standardized assessments. 

The 2004 math eurrieulum was Math Steps, a researeh-based, sequential program produeed by 
the Houghton-Mifflin publishing eompany, one of the major edueational publishing eompanies 
in the U.S. The eurrieulum was arranged so that teachers can follow a grade-specific, step-by- 
step plan that can be tailored to varying levels of achievement. Beyond the eore lesson. Math 
Steps provides worksheets and materials for teaehers to further assist students who are behind 
and also to provide additional, more ehallenging work for those who are advaneed. 

In 2005, bell’s literaey intervention ineluded a new eurrieulum — Houghton Mifflin’s Summer 
Success: Reading — a eurrieulum aligned with the No Child Left Behind legislation and the 
Reading Lirst program. This curriculum uses engaging, grade-level appropriate aetivities in the 
areas of fluency, comprehension, voeabulary, and word study. As a supplement to these 
programs, BELL has an extensive library with multieultural literature known as the BELL 
Reading Club. This library provides eulturally-relevant titles that engage seholars’ interest, 
provide on-level strueture to allow scholars success in reading and faeilitate development of pro- 
soeial values. Linally, BELL’s literacy approach included phonic instruction aligned to the 
reeommendations of the National Reading Panel. 

The changes to the literacy curriculum were driven largely by BELL’s desire to better address 
teaching and learning needs in the elassroom, ineluding providing resources for English 
Language Learners, ineorporating an explieit, integrated phonies eomponent, and ensuring 
adequate test preparation. In addition, the Summer Success: Reading eurrieulum provided 
additional supports important to the BELL program including better alignment to national and 
state learning standards. 

The BELL program also made a small ehange to its math eurrieulum, moving from Math Steps 
to Houghton Mifflin’s Summer Success: Math. Aligned with the National Council of Teachers of 
Mathematics (NCTM) standards, this math curricula provides grade-level appropriate activities 
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to give students practice in many areas of math, develop basic problem-solving strategies, and 
prepare students for state assessments. The BELL staff explained that the math curricula used in 
2004 and 2005 are very similar, but the 2005 curriculum is organized in a much more efficient 
way, laying out clear daily and weekly plans for teachers to follow. The 2005 math intervention 
also included more manipulatives and related activities to support mathematical development. 

The BELL program also provided a customization guide to teachers. This guide covered a 
number of issues ranging from assessment and homework to how to fit a five-day-a-week 
curriculum into BELL’s four-day instruction week. 

The BELL Program in Practice 

Program Characteristics 

Class Size 

The average class size in 2004 was nearly 19 students at the Boston site and 14 students at the 
New York site. In 2005, the class sizes were similar with an overall average of 19 again, 17 in 
the New York site and around 21 in each of the Boston sites. 

In 2004, class sizes ranged from as small as 13 students to as high as 25 students in Boston, and 

from 9 to 17 students in New York. This includes both the morning and afternoon classes. The 

range grew in 2005 going from a minimum of 9 to a maximum of 37 across the three sites with 

around 20 percent of the classes having more than 25 students each. BELL staff noted that this 

-1 

range is somewhat larger than normal because of the random assignment process. More 
importantly, however, each class had one teacher and at least one teaching assistant present, 
making the average student to adult ratio much lower. This ratio was 9.5 in Boston and 7 in New 
York in 2004. The student/adult ratios were similar in 2005 with an overall average of 8 and a 
range from a low of 4.5 to a maximum of 17 by classroom. Also the largest morning class (math 
or reading) had only 18 students in 2005. 

Staff Characteristics 

In 2004 all six of the New York teachers and 7 of 10 Boston teachers participated in our focus 
group discussions. In addition, all 17 of the Boston teaching assistants and all 6 of the New York 
assistants participated. Information sheets passed out prior to the focus groups asked participants 
to record the number of years they have served as a BELL teacher or teaching assistant, their 
school-year occupation, and the highest degree earned. BELL teachers who taught during the 
school-year were also asked the number of years they have been a teacher and the certifications 
they hold, if any. 

All teachers participating in the 2004 focus groups had either a Bachelor’s or graduate degree. 
All but 1 summer teacher is a current or former school-year teacher, and the group had an 



In particular, BELL did not replace students who dropped out of the program as much as they 
would normally as that might have interfered with the evaluation study. 
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average of roughly 7.3 years of teaching experience. Slightly over half of the teachers in the 
focus groups were certified (7 of the 13), while a majority (8 of the 13) had more than one year 
of experience working in the BELL program. 

Most of the teaching assistants in the BELL program in 2005 (17 of the 23) were college 
students pursuing degrees in education or related fields. Of the six not currently enrolled in 
college, three had already obtained a college degree and one was a retired teacher. Only two of 
the teaching assistants had neither obtained a college degree nor enrolled in college. Unlike the 
teachers, most of the teaching assistants were new to the program. Sixteen of the 23 teaching 
assistants were in their first summer of the BELL program. However, 18 of the 23 did serve as 
tutors in the BELL after-school program during the previous school year, making only 5 of the 
23 teaching assistants completely new to the BELL program. 

In 2005, 12 of the 14 New York teachers, 9 of the 14 Trotter teachers, and 12 of the 14 
Greenwood teachers participated in our focus group discussions. In addition, 13 of the 14 New 
York teaching assistants, 12 of the 14 Trotter teaching assistants, and 13 of the 14 Greenwood 
teaching assistants participated. Information sheets passed out prior to the focus groups asked 
participants to record the number of years they have served as a BELL teacher or teaching 
assistant, their school-year occupation, and the highest degree earned. BELL teachers who taught 
during the school-year were also asked the number of years they have been a teacher and the 
certifications they hold, if any. 

All but one teacher participating in the focus groups had either a Bachelor’s or graduate degree. 
With the exception of one paraprofessional and three substitute teachers, all summer teachers 
were also school-year teachers. The group had an average of roughly 4.9 years of teaching 
experience. Only five of the 33 teachers participating in the focus group lacked some kind of 
certification. None of the focus group participants had ever taught in the BELL summer program 
before. 

Most of the teaching assistants in the BELL program (25 of the 38) were college students. Of the 
13 not currently enrolled in college, all but one held either a Bachelor’s (eight participants) or an 
Associate (three participants) degree. Only one of the teaching assistants had not gone beyond 
high school education. Like the teachers, most of the teaching assistants were new to the 
program. Thirty-three of the 38 teaching assistants were in their first summer of the BELL 
program. However, 22 of the 33 did serve as tutors in the BELL after-school program during the 
previous school year, making 11 of the 38 teaching assistants completely new to the BELL 
program. 

Student Attendance Rates 

As noted earlier, the BELL 2004 summer program was open for 27 days in Boston and 25 in 
New York. We estimate that students were given around two hours of reading and one hour of 
math on approximately 20 of these days (Monday through Thursday), for a total of 40 and 20 
hours of reading and math instruction, respectively. This excludes any additional academic 
activities that may have occurred during the afternoon enrichment periods. In 2005, the program 
was open for 28 days in two sites and 29 days in one site, meaning BELL students could receive 
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slightly more instruction (22 or 23 days). This translates into around 44 hours of literacy 
instruction and 22 hours of math instruction over the summer. 

BELL staff provided us with attendance rolls by classroom for the days we attended their 
program (7/27/04 and 7/25/05 in NYC and 8/3/04, 7/26/05, and 7/27/05 in Boston). We observed 
slightly more students in attendance in the classrooms we observed than BELL reported (165 vs. 
160 in 2004 and 208 vs. 200 in 2005).^ 

BELL staff also provided us with data on attendance rates by day for all of their students across 
the full summer for both 2004 and 2005. Those data showed that the programs were closed for 
two days during the first week in Boston and for four days during the third and fourth weeks in 
New York in 2004. These disruptions may have lowered student attendance in subsequent days. 
In 2005, the site that had an extra day started one day earlier than the other two sites. Lor the 
remaining days in 2004, the data suggest that attendance rates among those who attended the 
program at all averaged around 74 percent in Boston and 64 percent in New York. In 2005, the 
rates were again generally in this range, except that at one site, the rates dropped to around 15 
percent during the last week.^ The 2005 rates were around 76 and 69 percent in the first two sites 
and around 65 percent over the entire summer at the site with low attendance during the final 
week. It is also interesting to note that attendance was generally lowest on Lridays suggesting 
that many parents valued the academic activities more than the non-academic ones. 

These rates are lower than those typical during the school-year (usually well over 80 percent) but 
much higher than for many out-of-school time programs. Also it should be noted that these rates 
include students who dropped out after the program started. 

Focus Groups of Teachers and Teaching Assistants 

Summer 2004: In 2004, the teachers and teaching assistants participating in New York and 
Boston focus groups discussed numerous aspects of the program including the program’s 
philosophy and curricula, as well as the challenges faced in implementing the BELL model. 
Morale among the two groups was quite high in both sites, with both teachers and teaching 
assistants discussing at length their high regard for the program. Among teachers, smaller class 
sizes and increased discretion in the classroom provided a welcome change from their 
experiences with the public school system. Most believed that the increased flexibility made 
them more effective teachers. Teaching assistants emphasized the program’s nurturing 
philosophy and emphasis on treating the whole child as fundamental strengths of the program. 



^ These numbers apply only to the classrooms where we obtained attendance data (12 classrooms 
in 2004 and 15 in 2005). The counts by classroom were very close in all but seven of these 27 
classrooms. The remaining discrepancies between our observations and the BELL rolls (one 
positive and one negative in 2004 and all positive but small in 2005) could be due to students 
coming and going during the day. 

^ In 2005, over 90 percent of those students who attended at all were in the program for at least 8 
days and 75 percent were there for at least 16 days. 
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In discussing the curricula used by BELL, nearly all the teachers in both sites spoke positively 
about the Voices of Love and Lreedom (VLL) reading and writing curriculum. The focus group 
participants felt the curriculum’s emphasis on diversity and culturally-appropriate issues were 
particularly relevant to the students’ lives. As one teacher noted, “[The] stories are about them or 
someone they know, and the pictures are of people who look like them.” Participants thought 
that these characteristics of VLL worked to keep students engaged, which increased its 
effectiveness as a teaching tool. Those who offered criticisms of VLL noted that, while the 
curriculum encourages free thinking about important concepts, there was not enough focus on 
mechanics. Some teachers felt, for example, that generally there were not enough writing 
exercises and more specifically, that fundamentals of grammar and sentence structure were not 
adequately addressed. 

Participant opinions about the Math Steps curriculum were more mixed, with opinions varying 
quite dramatically across the two sites. Compared to the Boston participants. New York 
participants were not nearly as positive about the math curriculum. New York participants liked 
the basic structure of the Math Step lessons, but thought that they were generally uncreative, 
especially in comparison to VLL. Boston teachers, however, liked Math Steps because of its 
emphasis on concrete lessons and real-life examples involving money and time. One potential 
explanation for the dramatic differences between the Boston and New York participants may 
involve the fact that the Boston public school system has instituted the TERC teaching method 
for elementary grade math, which does not focus on standard computational skills, but instead on 
making connections between mathematical ideas. ^ Some Boston teachers indicated that they 
welcomed the fundamentals-based structure of the Math Steps curriculum compared to the 
TERC method. 

The groups also discussed a number of challenges in implementing the BELL program. The 
issue most often discussed involved meeting the needs of all students, who varied widely in 
academic performance. Participants mentioned that while they wished BELL could provide 
“resource rooms” for students in need of remedial assistance, the BELL philosophy emphasizes 
keeping all students in the classroom. As such, teachers found that while the teaching assistants 
were quite helpful with these issues, it was still difficult to retain the attention and enthusiasm of 
students with a range of capabilities within one classroom. 

Teachers also mentioned the challenges of dealing with behavior problems in the classroom and 
the extent to which these problems took away from instruction time. While BELL explicitly 
recruits students with academic needs and the vast majority of learning disabilities can be served 
in their program, there are some severe special needs (including some emotional disturbances 
and some students with violent behavior) that they do not have the resources to serve. BELL 
staff typically refer these students to alternative programs that can better meet their needs. 
However, teachers in Boston who had been a part of the BELL program for multiple years noted 
that the 2004 group of children were the most challenging they had faced and one teacher noted 
that she often found herself "in the role of psychiatrist, parent and teacher." Teachers attributed 
this issue to the evaluation’s random assignment of program participants, which bypassed 



^ TERC math is a math curriculum developed by a nonprofit education research and development 
firm in Cambridge, MA. See http://www.terc.edu/ for more details. 
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bell’s normal screening procedures. Some teachers went as far as to say that the current 
evaluation was not truly an evaluation of the effects of the BELL program because the screening 
process was the program’s key to success. This was not mentioned as an issue among New York 
teachers, however. 

National level BELL staff say that they are not aware of any noticeable changes in the type of 
students served during the summer of 2004 compared to earlier years. However, these issues 
were also discussed in the teaching assistant focus group in Boston. Teaching assistants who had 
been at BELL for multiple years also noted that the program had screened for behavior problems 
in the past and that they could see a difference in the children this summer. Teaching assistants 
discussed that the program could not serve those children well because they lacked the resources 
to deal with many of the medical and behavioral issues encountered, including ADD and ADHD. 
Teaching assistants noted that they were not given specific information about students’ behavior 
disorders, which made it very difficult to know how to handle them properly. One participant 
stated, “At the very least, we need the lEPs [Individual Education Plan] for these kids so we 
know what is wrong with them.” Another added, “We’re drowning with all these issues.” 

Two other issues mentioned by focus group participants included poor communication between 
the teaching staff and executive office, and BELL’s lack of a science curriculum. However, the 
national office staff noted that direct communication with program staff is not one of their 
goals — rather, program staff are managed directly by site-level directors. 

Summer 2005: Morale and regard for the program were also high among teachers and teaching 
assistants in the summer of 2005. Teachers again mentioned the benefits of BELL’s smaller class 
sizes, which provided an increased opportunity for “quality mentoring” of students. All focus 
group participants appreciated BELL’s balanced approach of providing both academic and 
enrichment activities during the day. They felt that this approach addressed the needs of the 
whole child with a specific focus on self-esteem, and provided art and music activities that have 
been “lost” during the academic school year. 

As discussed, BELL changed its math and literacy curricula in 2005, and as a result, the nature of 
the focus group discussion about curricula changed somewhat as well. Overall, teachers spoke 
highly of the new literacy curriculum, focusing on how much the children enjoyed the lessons. 
The teachers also appreciated the curriculum’s “very well-designed” strategies for teaching 
literacy — visualization, prediction and games — and how they attempted to encourage the 
children to make a personal connection with the material. However, teachers noted that the 
literacy materials provided by Houghton Mifflin were theme-based and did not address 
multicultural issues. While BELL did provide supplemental materials designed for this purpose, 
these materials were not readily aligned with the primary curriculum and teachers found it 
difficult to “blend” them. In addition, teachers often found themselves running out of time and 
not being able to fit in the multicultural readings. Some teachers described incorporating both the 
curriculum and the supplement as a “challenge,” especially considering that they were not 
trained to do so. This was an issue because they new that this was something that the BELL 
program expected of them. 
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Similar to the 2004 focus groups, opinions were also mixed about the math curriculum in 2005. 
Staff at both Boston sites expressed concern about the amount of material covered with staff at 
one site noting that the math curriculum consisted of two-hour lessons that they had to fit into 
one hour. Staff in the other Boston site listed a variety of additional complaints. They felt that the 
curriculum was too scripted and focused on getting the answer right instead of how the answer is 
derived. Given that the curriculum required them to rush, many teachers in this site felt that they 
needed to forge ahead without allowing the children to master all of the objectives. 

In 2005, teachers in the New York site and second Boston site described the math curriculum 
positively. They noted that it attempted to incorporate a number of math components — geometry, 
operations, numbers, and more — in every lesson, that it provided an “organized structure of 
learning” for math concepts, and that it provided a math foundation that the children could 
understand. Compared to the math instruction that children received in the New York public 
schools, some teachers thought the curriculum was much more engaging and interesting from the 
child’s perspective. 

The focus groups indicated that a number of challenges existed in 2005 that were similar to the 
year before, but that they may pose less of an issue for the program. For example, teaching 
children with different abilities appeared to pose less of an issue compared to the summer of 
2004. Some teachers felt that the range of skill levels of the BELL students was relatively narrow 
and the curricula used addressed this range. Another group of teachers who thought the .skill 
range was wider mentioned that they received specific training on differentiated instruction to 
deal with the issue. Yet another group viewed the different skill levels as a strength where the 
higher achievers could model behavior for the other children. One set of teachers, however, did 
believe that there was a wide range of abilities and that the curricula did not allow for 
individualized teaching. Teachers in this group often felt that they had to move ahead even 
though many of the children did not understand the concepts. Some TAs also felt that the wide 
range of skill levels negatively affected the more advanced children who tended to be “bored out 
of their minds.” 

Behavioral problems among the students continued to be a problem in the summer of 2005 
although the strategies used to deal with this issue seemed to work fairly well. The program used 
a color coded “warning system” to alert children of their status. The system ranges from green to 
blue where green status indicates that a child is behaving well and blue means a call is going to 
be made to the child’s parents. 

Classroom Observations 

In all of the BELL sites observed in both 2004 and 2005, each classroom housed one grade-level 
of students, who remained in that classroom throughout the morning and received instruction in 
different subject areas. In 2004, each grade had three one-hour lessons: reading, writing, and 
math. In 2005, the reading and writing sessions were combined into one two-hour block followed 



n 

Indeed, national BELL staff informed us that in 2005 BELL added a three hour training module 
on how to differentiate instruction to meet a diverse range of student needs. 
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by one hour of math instruction. As part of the visits to each program site, a number of these 
BELL lessons were randomly selected for observation. Researchers rotated among classrooms 
observing the subjects being taught. In total, eight one-hour lessons were observed in New York 
and another eight were observed in Boston in 2004. In 2005, eight classrooms were again 
observed in New York. In Boston, five were observed at one site and three at another. 

The research team had two goals for the classroom observation. The first was to observe whether 
BELL teachers were properly implementing the BELL curricula. The second goal was to 
measure the level of students’ engagement in the classroom instruction. To meet these goals, the 
site visit team used two different observational protocols: one designed to capture teachers’ 
adherence to specific aspects of the prescribed curricula; and the other to measure the level of 
engagement of students in classroom activities. 

Teaching Practices 

Summers of 2004 and 2005: Results from the observations indicate that, overall, teachers at both 
sites and in both years were following the relevant BELL curricula and were well-equipped to 
effectively teach the BELL scholars. The vast majority of observed teachers were enthusiastic, 
caring, and able to manage their classrooms. All classes were held in traditional classrooms, 
which seemed spacious due to the small average class size. All of the lessons began with full- 
class participation, but students were usually given time to work independently later in the 
period. There was only a moderate amount of group work assigned and even when the children 
were allowed to work with partners, the activities did not generally seem to be collaborative by 
design. 

Academically, most of the classes followed lesson plans that were concurrent with the curricula. 
In all of the literacy lessons, stories were read or discussed and in some instances, writing 
exercises were assigned. Over half of the literacy classrooms observed participated in activities 
requiring the students to share personal experiences or relate the themes of the literature to their 
lives, a key component of the Voices for Love and Lreedom curriculum. The math classes were 
also, for the most part, consistent with the intended program. Often, the teacher would present a 
lesson to the whole class, hand out worksheets to be completed independently, and then be 
available for guided assistance as necessary. In other instances, students were given hands-on 
tasks, such as working with money, or participating in a competitive, math-related game. 

In 2004 the math periods seemed to be more disorganized than the reading periods and the 
children were less engaged. There are many reasons why this may be the case. One possibility — 
in both cities the math period is the third hour of academics, occurring right before the lunch 
break. In 2005 the math lessons did not appear to be less organized and students were no less 
engaged than in reading, but at two sites students had to collect their lunches during the math 
period creating a noticeable disruption in those lessons. 

During our focus groups and interviews we also learned about the role the teaching assistants 
(TAs) play in the BELL program. While students transition between different teachers for 
literacy, math, and the afternoon enrichment activities, they remain with the same teaching 
assistant throughout the day. This constant presence may facilitate the ability of the TA to 
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maintain discipline. In the morning this teaching assistant often works hand in hand with the 
teachers and, in at least one instance, was even helping to write lesson plans. In the afternoon, 
when children become more disruptive, the TAs tend to focus on the discipline issues. 

The consistency of the lessons plans was likely impacted by the training and the regular 
interaction of the teachers with senior staff. All of the TAs and Teachers had at least 32 hours of 

o 

training before the start of the program. In addition, during the summer the teachers had to 
submit their lesson plans for the following week each Friday to an instructional coordinator who 
would review them and provide comments. Staff also met on a weekly basis to discuss their 
curriculum and adjust it when needed. Finally, staff from the national office also conducted 
classroom observations. 

Student Engagement 

Summers of 2004 and 2005: In addition to observing teachers, the research team aimed to get a 
picture of the students’ engagement and the extent to which discipline issues interfered with the 
flow of the lesson. Regardless of the quality of the lesson or misbehavior that took place, the 
children appeared to be generally well supervised in both years. There was variation in behavior 
in different classrooms as well as during different academic lessons. Naturally, the students 
seemed to become more restless as the morning sessions wore on; however, the teachers and TAs 
employed successful management techniques. It appeared that the children were well aware of 
the behavior expectations and also of the standard repercussions. For the most part, teachers and 
teaching assistants used minimally disruptive methods to manage discipline, such as speaking to 
the child individually, using a “behavior chart” mounted on each wall, and reminding the child 
that misbehavior is simply negative attention. In 2004, it appeared that the primary role of the 
teaching assistant in each classroom was to handle discipline issues, which allowed the teacher to 
teach. Perhaps because of this, the teaching assistants seemed to be less enthusiastic than the 
teachers and slightly less patient. This seemed less true in 2005. In both years, most of the 
teachers were effective in reengaging students who were not paying attention, although in some 
classrooms, students were allowed to disengage if they were not disturbing others. 

National BELL staff noted that it is not the intention of the program model to have the teaching 
assistances focus on discipline issues. As with any of the patterns we observed, the staff behavior 
may have been affected by our presence. The national staff also informed us that they have 
added a training module called “collaborative teaching” where they communicate their 
expectations that the Teacher and TA will work together to leverage each other’s strengths and 
weaknesses. Also the TAs are supposed to support teaching and learning by working with small 
groups of students, leading whole group instruction at times, and giving scholars feedback on 
their work. 

To more precisely measure the engagement of BELL students, members of the site visit team 
used an observational matrix to record the activities of students during their literacy and math 
classes at 10-minute intervals throughout designated one-hour class periods. We observed a total 
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Some staff reported receiving 38 to 40 hours of training and one staff member even reported 
feeling that the BELL training was better than the regular summer school staff training in their 
city. 
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of 263 time intervals across both sites and years. About 61 percent of the observations were from 
2005 and they were about evenly split between Boston and New York.^ Trained observers 
recorded the percentage of students in class doing one of a number of activities at each time 
interval, including both positive activities (listening to teacher lecture, working independently, 
etc.) and negative (daydreaming, misbehaving, etc). 

Across all of these observations, 86 percent of BELL students, on average, were positively 
engaged in an academic activity at the observation points. On average, 33 percent of children 
were listening to a teacher lecture, while 22 percent were engaging in class discussion and 21 
percent were working independently. Another 10 percent where working productively together in 
small groups. Only 2 percent of the children were misbehaving while 5 percent were engaged in 
other, non-academic activities (going to and from the bathroom, etc.) and 7 percent were 
disengaged for other reasons (i.e., day dreaming). We found no evidence of statistically 
significant differences in these numbers across subjects or program years. 

Barriers to Implementation 

Obtaining and managing the physical space. 

Each year, the BELL program must obtain a space to house its summer program. Obtaining, 
negotiating, and maintaining this space is one of most difficult aspects of running the summer 
program. In New York, for example, program staff noted that it is unclear until late in the spring 
which New York schools will be available to use as a program site. This issue creates a delay in 
planning and in the program’s outreach efforts for the summer program. Administrators noted 
that they must negotiate every aspect of the program with the school principal, including 
accessible and off-limits areas within the building, drop-off, dismissal, and pick-up procedures, 
the use of air conditioners, and numerous other issues. In addition, working with school staff not 
under the purview of the BELL program — custodians, food service help, etc. — can also prove 
challenging. 

Language Issues 

BELL staff also noted that language barriers can be an issue, especially among those families 
who do not speak English or Spanish. While most students are proficient in English, 
communicating with parents who speak languages other than English or Spanish can be 
challenging. Often, students are used as translators, but these students are sometimes reluctant to 
translate information about their own school performance or behavioral issues. This is a 
relatively minor issue in the current BELL sites because most BELL families speak either 
English or Spanish and BELL has a number of Spanish speaking staff. This barrier is important 
to keep in mind, however, if one were to consider replicating the BELL program in additional 
sites. Indeed, in Boston there were a number of Vietnamese speakers. On the other hand, if an 



^ We did additional observations of the afternoon activities but those are not reported here. 
The standard errors for these numbers are all under 3 percentage points. 

We hired a Vietnamese speaker to conduct interviews with these parents in 2004 for the 
impact component of our study. 
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area has a larger number of students that speak another language, it is likely that there will also 
be a larger pool of possible staff people who speak that language. 

Transportation 

Unlike BELL’s school-year program, which exists at many schools in each city, the BELL 
summer program operates from just a few central locations and draws students from throughout 
each city. While the sites chosen are generally close to public transportation, getting to and from 
the program was an issue for at least some BELL families. In Boston, administrators noted that 
parents had the option of paying for transportation provided by the program. No direct mention 
of transportation was made in New York City. However, administrators noted that the 4:30 
pickup time in New York was problematic for many families because it was difficult for them to 
get off of work and make it to the program by that time. On the other hand, it should be noted 
that many summer programs operate for only four hours per day making transportation even 
more difficult for working parents. 

Variation across Sites 



In discussions with the BELL administration, two major implementation tensions surfaced 
pertaining to replicability. Lirst is the challenge of promoting program growth while also 
maintaining consistency. Second is the difficulty in implementing the BELL model with fidelity 
while adapting the program to meet local needs. It is important to the program that each site 
meets the rigorous standards of the BELL model, but as one administrator noted, “we struggle 
with consistent execution across sites every time.” Ideally, the organization strives to generate 
equally effective programs in each city and to have each child making similar academic gains. 
However, as new BELL after-school and summer sites take form, the leaders realize that many 
of the locations are exceptional while others still need to smooth out some operational kinks. As 
an organization wishing to replicate a specific curriculum, the BELL administration understands 
that adaptation is imperative as they expand; nevertheless, these alterations to the program model 
should not affect their high academic standards or their expectations for scholars. 
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V. Conclusions 



This report summarizes results from a process study of the BELL Summer Learning Program. 

We focus on three sets of issues — program fidelity, program replicability, and implications for 
our impact study. 

Program Fidelity 

Overall, our observations suggest that the BELL staff were implementing the program with 
fidelity to the model. The teachers appeared to be following their curricula in both reading and 
math and exhibiting good classroom management skills. This is not surprising since student/staff 
ratios were low, almost all of the teachers had experience teaching during the school-year (with 
an average of over seven years), and most of the teachers had at least some experience teaching 
in the BELL program. 

On the other hand, there were certain other aspects observed during our site visits that might 
temper or erode the impact of the program. These include issues related to student engagement, 
student attendance, teacher certification, and possible changes in the population served to 
facilitate the experimental design. While we found that student engagement was fairly high, 
about 14 percent were not engaged at a given point in time during the academic class-periods we 
observed. Attendance rates were probably higher than for many out-of-school time programs, but 
on average, well over a quarter of the students enrolled were not in attendance. The staff were 
generally well trained and experienced. Only about half of the teachers we spoke to were 
certified in 2004 but this number was up to 85 percent in 2005. Linally, some BELL staff 
reported that the children served in the summer of 2004 had more behavioral and attention 
problems than in previous years due to changes in BELL’s screening process, which they 
attributed to the random assignment of applicants. This issue was not seen as a problem in 2005. 

Program Replicability 

In the area of program replicability, we found both reasons to be optimistic and reasons for 
concern. On the one hand, the staff at the national and local levels appear to be highly qualified, 
energetic, and committed to this work and successful in maintaining a funding stream. Their 
involvement in this study and the fact that they have worked to make BELL an accredited 
provider of Supplemental Education Services (SES) in New York and Boston suggests that they 
are getting much of the needed ground work done to expand their program. On the other hand, it 
appears that space is quite difficult to obtain even in the sites where BELL is already operating. 
Ideally BELL would be able to take advantage of public schools that are often underutilized 
during the summer but this has proven difficult (though not impossible) to arrange. In addition, 
while BELL has been very successful at getting SES funding for their afterschool program run 
during the regular school year, SES funding has not generally been made available for summer 
learning programs. 




Implications for Impact Study 



Our impact study focused on reading rather than math in part because we expected somewhat 
larger impacts in reading than math for a number of reasons. First, previous evidence suggests 
that low-income students lose more in reading than math during the summer relative to their 
higher- income counterparts. Second, BELL offers more hours of reading instruction. Third, 
reading was offered first in the morning, while math was second so students may have been more 
energetic and consequently learned more during the reading classes. The likelihood of finding 
larger impacts in reading than in math is further supported by our site visits. In 2005, math 
classes in two of the three sites in our study had to be interrupted each day so that the students 
could collect their lunches. The impact report did, in fact, find positive impacts on reading test 
scores (Chaplin and Capizzano 2006). In future research we hope to investigate whether or not 
similar impacts might be found for math. 

Our earlier research showed that the BELL program improved summer learning among a group 
of low-income children. In this study we describe the components of this effective summer 
learning program. It is probably safe to assume that some of these components are necessary-for 
example reasonably good attendance rates of students. In other cases, however, it is more 
difficult to know whether or not the component identified mattered. Lor example, our results 
should not be used to suggest that the low child/staff ratio used at BELL is either necessary or 
ideal. Indeed, it might be more effective to have a higher ratio and use the extra funding to 
provide even more staff training. Lurther research on the impacts of variations of summer 
learning programs could help to better identify the optimal balance between these components 
and to thereby improve future efforts to reduce the summer learning gap. 
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